Line-of-movement generating apparatus, line-of-movement generating method, and non-transitory computer readable medium

ABSTRACT

A line-of-movement generating apparatus includes an acquisition unit, a detection unit, an extraction unit, a setting unit, a calculation unit, and a confirmation unit. The acquisition unit acquires images captured by an image capturing device at respective multiple time points. The detection unit detects one or more moving objects from the images. The extraction unit extracts a line of movement of each of the detected one or more moving objects. The setting unit sets a region where a line of movement tends to have a missing portion in an image captured by the image capturing device. The calculation unit calculates a likelihood indicating a degree of certainty of the extracted line of movement of each of the one or more moving objects. The confirmation unit confirms a line of movement of the one or more moving objects on the basis of the calculated likelihood.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is based on and claims priority under 35 USC 119 fromJapanese Patent Application No. 2016-061560 filed Mar. 25, 2016.

BACKGROUND

(i) Technical Field

The present invention relates to a line-of-movement generatingapparatus, a line-of-movement generating method, and a non-transitorycomputer readable medium.

(ii) Related Art

There is known a technique that captures images of an open space in acommercial facility or the like, and by using the captured images,analyzes lines of movement of persons or counts the number of visitors.

SUMMARY

According to an aspect of the invention, there is provided aline-of-movement generating apparatus including an acquisition unit, adetection unit, an extraction unit, a setting unit, a calculation unit,and a confirmation unit. The acquisition unit acquires images capturedby an image capturing device each at a corresponding one of multipletime points. The detection unit detects one or more moving objects fromthe images each captured at a corresponding one of multiple time points.The extraction unit extracts a line of movement of each of the detectedone or more moving objects. The setting unit sets a region where a lineof movement tends to have a missing portion in an image captured by theimage capturing device. The calculation unit calculates a likelihoodindicating a degree of certainty of the extracted line of movement ofeach of the one or more moving objects, the calculation unit calculatingthe likelihood according to an expression by which the likelihooddecreases as a starting point or an ending point of the line of movementis further from the region. The confirmation unit confirms a line ofmovement of the one or more moving objects on the basis of thecalculated likelihood.

BRIEF DESCRIPTION OF THE DRAWINGS

An exemplary embodiment of the present invention will be described indetail based on the following figures, wherein:

FIG. 1 illustrates how a person moves;

FIG. 2 illustrates a process related to detection of a person andextraction of a line of movement in which the person moves from capturedimages;

FIG. 3 illustrates a process related to extraction of lines of movementin which persons move;

FIG. 4 is a block diagram illustrating a configuration of aline-of-movement generating apparatus according to an exemplaryembodiment of the present invention;

FIG. 5 is a flowchart illustrating a process related to setting of amissing region, the process being performed by the line-of-movementgenerating apparatus according to the exemplary embodiment;

FIGS. 6A and 6B illustrate a specific exemplary process related tosetting of missing regions according to the exemplary embodiment;

FIGS. 7A, 7B, and 7C illustrate a specific exemplary process related tosetting of missing regions according to the exemplary embodiment;

FIG. 8 is a flowchart illustrating a process of generating a line ofmovement, the process being performed by the line-of-movement generatingapparatus according to the exemplary embodiment on the basis of themissing region;

FIGS. 9A and 9B illustrate a specific exemplary process for generating aline of movement on the basis of missing regions according to theexemplary embodiment; and

FIG. 10 is a flowchart illustrating a process related to setting of amissing region, the process being performed by the line-of-movementgenerating apparatus according to a modification of the presentinvention.

DETAILED DESCRIPTION

The present invention is an invention for generating a line of movementof a moving object present in an open space in a commercial facility orthe like on the basis of captured images of the open space. Although thecase where the moving object is a person will be described below, thepresent invention may be applied to generation of a line of movement ofa moving object other than a person (e.g., animal).

Next, an outline of a process of generating a line of movement accordingto an embodiment of the present invention will be described.

FIG. 1 illustrates how a person H present in an open space moves. InFIG. 1, positions where the person H is present at respective timepoints T1, T2, and T3 when seen from above are illustrated. FIG. 2illustrates a process related to detection of the person H andextraction of a line of movement of the person H from captured images ofthe open space in which the person H is present. In FIG. 2, capturedimages IM1, IM2, and IM3 obtained at the time points T1, T2, and T3,respectively, are illustrated. Each of the captured images is a stillimage of a single frame included in a moving image.

The process for generating the line of movement according to theexemplary embodiment is broadly divided into “person detection”,“line-of-movement extraction”, “likelihood calculation for each line ofmovement”, and “line-of-movement-likelihood calculation”.

Firstly, a person is detected from the captured images in thechronological order of the time points for image capturing. Here, theperson H is detected from the captured image IM1 first. Detection of aperson may be performed according to a known algorithm, such as abackground subtraction method or a method usinghistograms-of-oriented-gradient (HOG) feature amounts. From the capturedimage IM1, a person region A1 is detected. In the exemplary embodiment,a person region is defined by a circumscribed rectangle of a region of acaptured person. The person region A1 has a dimension D₁. The dimensionof a person region is defined by, for example, the area of the personregion and may also be defined by the length of a side, the length of adiagonal, or the like. On the basis of the position of the person regionA1, a position P1 of the person H at the time point T1 is specified.Although the position of the person corresponds to the center of gravityof the person's head in the exemplary embodiment, the position maycorrespond to another position within the person region.

Then, the person H is detected from the captured image IM2. From thecaptured image IM2, a person region A2 is detected. The person region A2has a dimension D₂. On the basis of the position of the person regionA2, a position P2 of the person H at the time point T2 is specified.Then, the person H is detected from the captured image IM3. From thecaptured image IM3, a person region A3 is detected. The person region A3has a dimension D₃. On the basis of the position of the person regionA3, a position P3 of the person H at the time point T3 is specified.Here, the position P1 corresponds to the starting point of a line ofmovement M, and the position P3 corresponds to the ending point of theline of movement M.

Secondly, on the basis of the regions where the person has been detectedfrom the captured images, the line of movement of the person isextracted. The line of movement is specified by a combination of personregions detected from images captured at respective multiple timepoints. Here, by connecting the positions P1, P2, and P3 with a line inthe chronological order of the time points for image capturing, the lineof movement M of the person H is extracted.

Thirdly, the likelihood is calculated for each line of movement. Thelikelihood of a line of movement is a value indicating a degree ofcertainty of the extracted line of movement as a line of movement of aperson. A likelihood L_(T)(D) is calculated according to the followingExpression (1), for example.

$\begin{matrix}{{L_{T}(D)} = {{L_{D\; 1}\left( D_{1} \right)}{\prod\limits_{i = 2}^{n}{L_{Di}\left( D_{i} \right)}}}} & (1)\end{matrix}$

In Expression (1), n is a natural number and is determined according tothe number of person regions that form a single line of movement. In thecase of the line of movement M, n=3 is satisfied. L_(Di)(D_(i)) (i is anatural number) is a function including a dimension D_(i) of a personregion Ai as a variable. The dimension of the person region isconsidered to fall within a certain range according to the size of theperson's body. That is, the dimension of the detected person regionserves as an indicator for determining whether it is certain that aperson has been detected or whether noise or the like has beenerroneously detected as a person.

Specifically, when i=1 is satisfied, L_(Di)(D_(i)), i.e., L_(D1)(D₁), isa function including the dimension D₁ of the person region A1 as avariable. L_(D1)(D₁) increases as the certainty of the dimension D₁indicating the dimension of a person region increases, and L_(D1)(D₁)decreases as the certainty of the dimension D₁ indicating the dimensionof a person region decreases. Here, a condition 0≦L_(D1)(D₁)≦1 issatisfied. L_(D1)(D₁) may be a linear function, a quadratic function, ahigher-degree function, or another function.

When i≧2 is satisfied, L_(Di)(D_(i)) is a function including, asvariables, the dimension D_(i) of the person region Ai and a valueaccording to the positional relationship between the detected personregion Ai and a person region Ai−1. The positional relationship isspecified by, for example, the distance between the person region Ai andthe person region Ai−1 and the direction from one of the person regionAi and the person region Ai−1 to the other. When i≧2 is satisfied,L_(Di)(D_(i)) has a value according to the degree of certainty of thedimension D_(i) indicating the dimension of a person region and thedegree of certainty of the positional relationship between the personregion Ai and the person region Ai−1 indicating movement of a person.Here, a condition 0≦L_(Di)(D_(i))≦1 is satisfied. When i≧2 is satisfied,L_(Di)(D_(i)) may be a linear function, a quadratic function, ahigher-degree function, or another function.

Fourthly, a line-of-movement likelihood is calculated. The likelihoodL_(T)(D) indicates the degree of certainty of a single line of movement.However, in some cases, multiple lines of movement may be extracted fromcaptured images. Although the multiple lines of movement typicallyindicate lines of movement of multiple persons, depending on the case,noise included in the captured images may be extracted as a line ofmovement. Accordingly, the line-of-movement likelihood is calculated asa value indicating the degree of certainty of one or more lines ofmovement as a whole extracted from the captured images. Theline-of-movement likelihood is specifically a product of likelihoodsL_(T)(D) of one or more lines of movement. The line-of-movementlikelihood is equal to the likelihood L_(T)(D) in the case where asingle line of movement has been extracted. A larger value of theline-of-movement likelihood indicates a larger degree of certainty ofthe extracted one or more lines of movement.

In the exemplary embodiment, “line-of-movement extraction” is performedby random sampling using a Markov chain Monte Carlo method.Specifically, multiple sets of lines of movement are extracted bychanging at random, by using random numbers, the combination of personregions detected from the images captured at the respective multipletime points. Here, the case where the person region Ai has beenextracted as illustrated in the upper part of FIG. 3 will be discussed.In this part, each rectangle indicates a person region Ai. In thisillustration, multiple sets of lines of movement (see dashed lines inFIG. 3) having different combinations of person regions are extracted asillustrated in “first result of line-of-movement extraction”, “secondresult of line-of-movement extraction”, . . . in FIG. 3.

The processes “likelihood calculation for each line of movement” and“line-of-movement-likelihood calculation” are performed for each of themultiple sets of lines of movement. Specifically, upon calculation of aline-of-movement likelihood regarding a result of extraction of acertain line of movement, the line-of-movement likelihood is comparedwith the highest line-of-movement likelihood among previously calculatedline-of-movement likelihoods regarding results of extraction of otherlines of movement. Then, a result of line-of-movement extraction havinga higher line-of-movement likelihood is selected. Then, the combinationof person regions that form at least some of the lines of movement inthe selected result of line-of-movement extraction is changed at random,thereby calculating the line-of-movement likelihood again. Thecalculated line-of-movement likelihood is compared with the highestline-of-movement likelihood at this time point, and a result ofline-of-movement extraction having a higher line-of-movement likelihoodis selected. After this process is iterated a prescribed number oftimes, a result of line-of-movement extraction having the highestline-of-movement likelihood is eventually confirmed (determined) to beone or more lines of movement extracted from the captured images.

However, merely employing this method may fail to generate an originalline of movement in some cases. In these cases, a line of movement iscut in the middle (has a missing portion), although the line of movementis supposed to be a continuous line, owing to noise or other factorsincluded in the captured images. The line-of-movement generatingapparatus according to the exemplary embodiment has a configuration thataddresses this issue.

FIG. 4 is a block diagram illustrating a configuration of aline-of-movement generating apparatus 1 according to the exemplaryembodiment. The line-of-movement generating apparatus 1 includes acontroller 10, an operation unit 20, an interface 30, a communicationunit 40, a display unit 50, a missing-point-data storing unit 60, amissing-region-data storing unit 70, and a line-of-movement-data storingunit 80.

The controller 10 includes a processor including, for example, a centralprocessing unit (CPU), a read only memory (ROM), and a random accessmemory (RAM). The CPU controls each unit of the line-of-movementgenerating apparatus 1 by reading to the RAM a program stored in the ROMand executing the program.

The operation unit 20 includes a touch sensor or hardware key pad, forexample, and receives operations performed by a user. The interface 30is an interface to be connected to an image capturing device 100. Theimage capturing device 100 is installed in such a manner as to captureimages of an open space for which a line of movement is to be generated.The image capturing device 100 outputs, to the interface 30, capturedimages obtained by image capturing in the order the images were captured(in the order of frames), for example. The communication unit 40includes a modem, for example, and performs communication by beingconnected to a communication line, such as a line to the Internet. Thedisplay unit 50 includes a liquid crystal display, for example, anddisplays images on a display surface.

The missing-point-data storing unit 60, the missing-region-data storingunit 70, and the line-of-movement-data storing unit 80 are formed by oneor more memory devices (e.g., hard disk device).

The missing-point-data storing unit 60 stores missing point data. Themissing point data is data indicating the position where a line ofmovement is missing in a captured image, and is specifically dataindicating the position of the starting point or ending point of theline of movement.

The missing-region-data storing unit 70 stores missing region dataindicating a missing region. The missing region data is data unique tothe image capturing device 100 and is data indicating a region(hereinafter referred to as “missing region”) where a line of movementtends to be missing in an image captured by the image capturing device100.

The line-of-movement-data storing unit 80 stores line-of-movement dataindicating a confirmed line of movement. The line-of-movement data isused in order to count the number of visitors or for other purposes.

The controller 10 realizes functions corresponding to an acquisitionunit 11, a detection unit 12, an extraction unit 13, a calculation unit14, a confirmation unit 15, and a setting unit 16. The calculation unit14 further includes a first likelihood calculating unit 141 and a secondlikelihood calculating unit 142.

The acquisition unit 11 acquires, through the interface 30, the imagesobtained by the image capturing device 100 capturing images atrespective multiple time points. The detection unit 12 detects one ormore persons from the acquired images captured at the multiple timepoints. The extraction unit 13 extracts, on the basis of person regionswhere the detected one or more persons are present, a line of movementof each of the one or more persons. The extraction unit 13 extractsmultiple sets of lines of movement by changing the combination of theperson regions by random sampling using the Markov chain Monte Carlomethod.

The calculation unit 14 calculates a likelihood L_(T)(D) of each of theextracted lines of movement. The first likelihood calculating unit 141calculates the likelihood L_(T)(D) according to Expression (1). Thesecond likelihood calculating unit 142 calculates the likelihoodL_(T)(D) not according to Expression (1) but according to an expression(Expression (2) described later) by which the likelihood L_(T)(D)decreases as the starting point or ending point of a line of movement isfurther from a missing region.

On the basis of the likelihoods L_(T)(D) calculated by the calculationunit 14, the confirmation unit 15 confirms the line of movement of theone or more persons. For each of the extracted multiple sets of lines ofmovement, the confirmation unit 15 calculates a product of thelikelihoods L_(T)(D) as a line-of-movement likelihood and confirms aline of movement having the highest line-of-movement likelihood. Theconfirmation unit 15 causes line-of-movement data indicating theconfirmed line of movement to be stored in the line-of-movement-datastoring unit 80. In addition, on the basis of the starting point andending point of the confirmed line of movement, the confirmation unit 15records missing point data in the missing-point-data storing unit 60.

The setting unit 16 sets a missing region in an image captured by theimage capturing device 100. On the basis of the missing point datastored in the missing-point-data storing unit 60, the setting unit 16specifies distribution of missing points in the captured image and setsa missing region on the basis of the distribution. The setting unit 16records the missing region data indicating the missing region in themissing-region-data storing unit 70. On the basis of the missing regiondata stored in the missing-region-data storing unit 70, the secondlikelihood calculating unit 142 calculates the likelihood L_(T)(D).

FIG. 5 illustrates a process related to setting of the missing region,the process being performed by the line-of-movement generating apparatus1. FIGS. 6A, 6B, 7A, 7B, and 70 illustrate specific exemplary processesrelated to setting of missing regions.

The controller 10 acquires captured images from the image capturingdevice 100 through the interface 30 (step S1). FIG. 6A illustrates acaptured image IM obtained in the case where an image of certain openspace is captured from above in an obliquely downward direction. Thecaptured image IM includes walls W1 and W2 and pillars Ob1, Ob2, and Obias structures that may possibly affect a line of movement of a person.Hereinafter, any image captured by the image capturing device 100 isreferred to as a “captured image IM” without particularly distinguishingthe time points for image capturing.

Then, the controller 10 detects persons from the images captured atrespective multiple time points (step S2), the images having beenacquired in step S1. Then, on the basis of person regions where thedetected persons are present, the controller 10 extracts a line ofmovement of each of the persons (step S3). Here, it is assumed that aline of movement M1 of a person H1 and a line of movement M2 of a personH2 are extracted as illustrated in FIG. 6B.

Then, according to Expression (1), the controller 10 calculates alikelihood L_(T)(D) of each of the extracted lines of movement (stepS4). In the case of FIG. 6B, the controller 10 calculates a likelihoodL_(T)(D) of the line of movement M1 and a likelihood L_(T)(D) of theline of movement M2. Then, the controller 10 calculates, as aline-of-movement likelihood, a product of the likelihoods L_(T)(D) ofthe extracted one or more lines of movement (step S5). In the case ofFIG. 6B, the controller 10 calculates, as the line-of-movementlikelihood, a product of the likelihood L_(T)(D) of the line of movementM1 and the likelihood L_(T)(D) of the line of movement M2.

Next, the controller 10 determines whether or not the line-of-movementlikelihood has been calculated for a prescribed number of lines ofmovement (step S6). The prescribed number is a number determined inadvance and is “1000”, for example. If the determination in step S6 is“NO”, the controller 10 returns to step S3. Then, the controller 10extracts another line of movement by changing the combination of theperson regions. Then, the controller 10 calculates a likelihood L_(T)(D)according to Expression (1) and calculates a line-of-movementlikelihood. Step S3 through step S6 are performed by random samplingusing the above-described Markov chain Monte Carlo method.

Upon calculation of the line-of-movement likelihood for a prescribednumber of lines of movement, the controller 10 determines “YES” in stepS6. Then, the controller 10 confirms, as a line of movement of a person,a line of movement having the highest line-of-movement likelihood atthis time point (step S7). Here, it is assumed that the line of movementM1 of the person H1 and the line of movement M2 of the person H2 areconfirmed.

Then, on the basis of the starting point and ending point of theconfirmed line of movement, the controller 10 records missing point datathe missing-point-data storing unit 60 (step S8). Here, the missingpoint data recorded by the controller 10 includes missing points at thefollowing positions as illustrated in FIG. 7A: a position P11, which isthe starting point of the line of movement M1; a position P12, which isthe ending point thereof; a position P21, which is the starting point ofthe line of movement M2; and a position P22, which is the ending pointthereof.

Then, the controller 10 determines whether or not missing point dataregarding a prescribed number of missing points has been stored in themissing-point-data storing unit 60 (step S9). The prescribed number is anumber determined in advance and is “1000”, for example. In the casewhere the determination in step S9 is “NO”, the controller 10 ends theprocess in FIG. 5.

By iterating step S1 through step S8, the line-of-movement generatingapparatus 1 records (stores) missing point data in themissing-point-data storing unit 60.

When the number of missing points in the missing point data stored inthe missing-point-data storing unit 60 increases to reach the prescribednumber, the controller 10 determines “YES” in step S9 and proceeds tostep S10. Then, on the basis of the missing point data stored in themissing-point-data storing unit 60, the controller 10 specifiesdistribution of missing points (step S10). Here, the controller 10specifies a region having a relatively high density of missing points byperforming clustering on the basis of missing points plotted on acaptured image. The algorithm for clustering is, for example, k-meansclustering, minimum mean distribution, or nearest neighbor distance, butis not limited to a particular one.

Then, on the basis of the specified distribution of the missing points,the controller 10 sets, as a missing region, a regions where the line ofmovement tends to be missing in a captured image (step S11). In stepS11, the controller 10 records missing region data indicating the setmissing region in the missing-region-data storing unit 70.

Here, the case where the missing points are distributed as illustratedin FIG. 7B in the captured image IM will be discussed. In theillustration, each circle represents a missing point. In this case, thecontroller 10 sets missing regions C1, C2, and C3 as illustrated in FIG.7C, for example. Missing points are supposed to appear at and near adoorway or at and near obstruction, such as positions behind the pillarsOb1, Ob2, and Ob3, with a high frequency. In other words, missing pointsdo not appear with a high frequency in a region which is not at or nearthe doorway and not at or near obstruction, such as a region at and nearthe center of the captured image IM. As illustrated in FIG. 7B, missingpoints appearing in such a region, although the number thereof is small,are generated as a result of noise or the like included in the capturedimage. Accordingly, there is a low possibility that a missing pointappearing in such a region indicates the starting point or ending pointof an original line of movement of a person. For this reason, thecontroller 10 excludes, from missing regions, a region with no missingpoints and a region having a density of missing points that is lowerthan a fixed value.

The process performed by the controller 10 related to setting of themissing region has been described above.

FIG. 8 is a flowchart illustrating a process related to generation of aline of movement on the basis of a missing region, the process beingperformed by the line-of-movement generating apparatus 1.

The controller 10 acquires captured images from the image capturingdevice 100 through the interface 30 (step S21). Then, the controller 10detects persons from the images captured at respective multiple timepoints (step S22), the images being acquired in step S21. Then, on thebasis of person regions where the detected persons are present, thecontroller 10 extracts a line of movement of each of the persons (stepS23). Steps S21, S22, and S23 are the same as steps S1, S2, and 53,respectively.

Then, according to the following Expression (2), the controller 10calculates a likelihood L_(T)(D) of each line of movement (step S24).

$\begin{matrix}{{{L_{T}(D)} = {{L_{D\; 1}^{\prime}\left( D_{1} \right)}{\prod\limits_{i = 2}^{n}{L_{Di}\left( D_{i} \right)}}}}{{where}\mspace{14mu} {L_{D\; 1}^{\prime}\left( D_{1} \right)}} = {{L_{D\; 1}\left( D_{1} \right)}{L_{V}\left( D_{1} \right)}\mspace{31mu} {\left. L_{V} \right.\sim{N\left( {\mu_{j},\sigma_{j}^{2}} \right)}}}} & (2)\end{matrix}$

Expression (2) corresponds to an expression in which “L_(D1)(D₁)” inExpression (1) is replaced by “L′_(D1)(D₁)”. L′_(D1)(D₁) is a functionobtained by multiplying the function L_(D1)(D₁), which is used inExpression (1), by a function L_(V)(D₁). L_(V)(D₁) increases as thedistance decreases between the starting point of a line of movement anda point (e.g., the center of gravity) within a missing region that isthe closest to the starting point, and L_(V)(D₁) decreases as thedistance increases. L_(V)(D₁) is a function of normal distribution ofmeans μ_(j) and distribution σ_(j) ². However, L_(V)(D₁) is not limitedthereto and may be a linear function, a quadratic function, ahigher-degree function, or another function.

Then, the controller 10 calculates, as a line-of-movement likelihood, aproduct of the calculated likelihoods L_(T)(D) (step S25). Next, thecontroller 10 determines whether or not the line-of-movement likelihoodhas been calculated for a prescribed number of lines of movement (stepS26). If the determination in step S26 is “NO”, the controller 10returns to step S23. If the determination in step S26 is “YES”, thecontroller 10 confirms, as a line of movement of a person, a line ofmovement having the highest line-of-movement likelihood at this timepoint (step S27). Steps S25, S26, and S27 are the same as steps S5, S6,and S7, respectively.

If a likelihood L_(T)(D) of a certain line of movement calculatedaccording to Expression (2) is compared with a likelihood L_(T)(D)thereof calculated according to Expression (1), the likelihood L_(T)(D)calculated according to Expression (2) has a smaller value than thelikelihood L_(T)(D) calculated according to Expression (1) as thestarting point of the line of movement is further from a missing region.Accordingly, if the extracted line of movement has a starting point farfrom the missing region, the line-of-movement likelihood is also low,and there is a low possibility that the line of movement is eventuallyconfirmed as a line of movement of a person.

Here, the case will be discussed where a line of movement M21 and a lineof movement M31, which are illustrated in FIG. 9A, are extracted in stepS23. The line of movement M21 is a line of movement connecting theposition P21 and the position P22. The line of movement M31 is a line ofmovement connecting a position P31, a position P32, and a position P33.As a result of calculation according to Expression (1), there is apossibility that the line of movement M21 and the line of movement M31are eventually confirmed. However, as a result of calculation accordingto Expression (2), there is a possibility that the line of movement M21and the line of movement M31 do not represent an original line ofmovement because the position P31, which is the starting point of theline of movement M31, is relatively far from the missing region C1.

In contrast, in the case of FIG. 9B, a line of movement M22 connectingthe positions P21, P22, P31, P32, and P33 is extracted. As a result ofcalculation according to Expression (2), there is a possibility that theline-of-movement likelihood in the case where the line of movement isextracted as illustrated in FIG. 9B is higher than that in the casewhere the line of movement is extracted as illustrated in the FIG. 9A.This is because the position P21, which is the starting point of theline of movement M31, is located within the missing region C1. In thismanner, it is considered that there is a relatively high possibilitythat the line of movement having a starting point within a missingregion is an original line of movement of a person. As a result ofcalculation according to Expression (2), the possibility of confirmingthe line of movement M22, not the line of movement M21 or the line ofmovement M31, is higher than in the case of calculation according toExpression (1), and accordingly, a line of movement having a higherdegree of certainty as a line of movement of a person is more likely tobe confirmed.

The process performed by the controller 10 related to generation of aline of movement on the basis of a missing region has been describedabove.

As described above, according to the line-of-movement generatingapparatus 1, a line of movement of a moving object is accuratelygenerated as compared with the case where a line of movement of a movingobject is generated by not taking into account a region where a line ofmovement tends to be cut. In addition, on the basis of the result ofactual generation of a line of movement, the line-of-movement generatingapparatus 1 sets a missing region according to an open space.

The present invention may be implemented in an exemplary embodimentdifferent from the above-described exemplary embodiment. In addition,modifications described below may be combined with each other.

The line-of-movement generating apparatus 1 may set a missing regionthat has been designated through a user operation. FIG. 10 is aflowchart illustrating a process related to setting of a missing region,the process being performed by the line-of-movement generating apparatus1.

The controller 10 acquires a captured image from the image capturingdevice 100 through the interface 30 and displays the captured image onthe display unit 50 (step S31). The captured image displayed on thedisplay unit 50 is, for example, the same as the captured image IMillustrated in FIG. 6A.

Then, in the state where the captured image is displayed on the displayunit 50, the controller 10 receives a user operation of designating amissing region through the operation unit 20 (step S32). A user visuallyconfirms the image displayed on the display unit 50 and designates themissing region by operating the operation unit 20. Then, the controller10 sets the missing region designated through the operation unit 20(step S33). That is, the controller 10 records missing region data inaccordance with the user operation in the missing-region-data storingunit 70.

By using a configuration in which the user designates the missingregion, the line-of-movement generating apparatus 1 sets the missingregion without using a result of actual generation of a line ofmovement.

The line-of-movement generating apparatus 1 may select a method forsetting the missing region depending on the mode.

The line-of-movement generating apparatus 1 may generate a line ofmovement on the basis of a captured image obtained by a wide-anglecamera, such as a fish-eye camera, i.e., a captured image includingdistortion caused by the imaging lens. In the exemplary embodiment ofthe present invention, specs of the image capturing device are notlimited to particular ones.

The line-of-movement generating apparatus 1 may extract a line ofmovement without random sampling using a Markov chain Monte Carlomethod.

Expression (2) is merely an exemplary expression for calculating alikelihood of a line of movement. It is sufficient for theline-of-movement generating apparatus 1 to calculate the likelihoodaccording to an expression by which the likelihood decreases as thedistance from a missing region increases. In addition, according toExpression (2), a likelihood in accordance with the distance between thestarting point of a line of movement and a missing region is calculated.However, the line-of-movement generating apparatus 1 may calculate alikelihood in accordance with the distance between the ending point of aline of movement and a missing region or the distance between each ofthe starting point and the ending point of a line of movement and amissing region.

The line-of-movement generating apparatus 1 may output, by displaying,transmitting, printing, or another method, data indicating a confirmedline of movement in place of or in addition to recording it in theline-of-movement-data storing unit 80.

The hardware configuration and functional configuration of theline-of-movement generating apparatus 1 are not limited to thosedescribed above in the exemplary embodiment.

Each function realized by the controller 10 of the line-of-movementgenerating apparatus 1 according to the above-described exemplaryembodiment may be implemented by one or more hardware circuits, by anarithmetic device executing one or more programs, or by a combinationthereof. In addition, in the case where the functions of the controller10 are implemented by using programs, the programs may be provided bybeing stored in a computer readable recording medium, such as a magneticrecording medium (e.g., magnetic tape or magnetic disk (e.g., hard diskdrive (HDD) or flexible disk (FD)), an optical recording medium (e.g.,optical disc), a magneto-optical recording medium, or a semiconductormemory, or may be distributed through a network. In addition, thepresent invention may be provided as a computer implementedline-of-movement generating method.

The foregoing description of the exemplary embodiment of the presentinvention has been provided for the purposes of illustration anddescription. It is not intended to be exhaustive or to limit theinvention to the precise forms disclosed. Obviously, many modificationsand variations will be apparent to practitioners skilled in the art. Theembodiment was chosen and described in order to best explain theprinciples of the invention and its practical applications, therebyenabling others skilled in the art to understand the invention forvarious embodiments and with the various modifications as are suited tothe particular use contemplated. It is intended that the scope of theinvention be defined by the following claims and their equivalents.

What is claimed is:
 1. A line-of-movement generating apparatuscomprising: an acquisition unit that acquires images captured by animage capturing device each at a corresponding one of a plurality oftime points; a detection unit that detects one or a plurality of movingobjects from the images each captured at a corresponding one of theplurality of time points; an extraction unit that extracts a line ofmovement of each of the detected one or plurality of moving objects; asetting unit that sets a region where a line of movement tends to have amissing portion in an image captured by the image capturing device; acalculation unit that calculates a likelihood indicating a degree ofcertainty of the extracted line of movement of each of the one orplurality of moving objects, the calculation unit calculating thelikelihood according to an expression by which the likelihood decreasesas a starting point or an ending point of the line of movement isfurther from the region; and a confirmation unit that confirms a line ofmovement of the one or plurality of moving objects on the basis of thecalculated likelihood.
 2. The line-of-movement generating apparatusaccording to claim 1, wherein the extraction unit extracts a pluralityof sets of lines of movement by changing a combination of moving objectsdetected from the images each captured at a corresponding one of theplurality of time points, and wherein, among the plurality of sets oflines of movement, the confirmation unit confirms a line of movementhaving a largest product of likelihoods of the extracted one orplurality of lines of movement.
 3. The line-of-movement generatingapparatus according to claim 1, wherein the setting unit sets the regionon the basis of distribution of a starting point or an ending point ofthe confirmed line of movement of the one or plurality of movingobjects.
 4. The line-of-movement generating apparatus according to claim1, wherein the setting unit sets the region on the basis of an operationfor designating the region.
 5. A line-of-movement generating methodcomprising: acquiring images captured by an image capturing device eachat a corresponding one of a plurality of time points; detecting one or aplurality of moving objects from the images each captured at acorresponding one of the plurality of time points; extracting a line ofmovement of each of the detected one or plurality of moving objects;setting a region where a line of movement tends to have a missingportion in an image captured by the image capturing device; calculatinga likelihood indicating a degree of certainty of the extracted line ofmovement of each of the one or plurality of moving objects, thelikelihood being calculated according to an expression by which thelikelihood decreases as a starting point or an ending point of the lineof movement is further from the region; and confirming a line ofmovement of the one or plurality of moving objects on the basis of thecalculated likelihood.
 6. A non-transitory computer readable mediumstoring a program causing a computer to execute a process for generatinga line of movement, the process comprising: acquiring images captured byan image capturing device each at a corresponding one of a plurality oftime points; detecting one or a plurality of moving objects from theimages each captured at a corresponding one of the plurality of timepoints; extracting a line of movement of each of the detected one orplurality of moving objects; setting a region where a line of movementtends to have a missing portion in an image captured by the imagecapturing device; calculating a likelihood indicating a degree ofcertainty of the extracted line of movement of each of the one orplurality of moving objects, the likelihood being calculated accordingto an expression by which the likelihood decreases as a starting pointor an ending point of the line of movement is further from the region;and confirming a line of movement of the one or plurality of movingobjects on the basis of the calculated likelihood.